1. Field of Invention
The present invention relates generally to machine vision, and more particularly, to a method of modeling the off-specular reflection of light to allow a machine to recognize the image of an object illuminated by varying unknown lighting conditions.
2. Description of Related Art
It is clear that changes in lighting can have a large effect on how an object, such as a person, looks. Understanding this is essential to building effective object recognition systems.
While the Lambertian (or uniform) model of the scattering of light from real surfaces is central to any study of object illumination, more sophisticated modeling is essential. In general, a variety of conditions combine to introduce anomalies into the simple model, especially those related to specular reflection (strong scattering over a narrow range of angles) and geometrically induced shadows. Recently, consideration has been given to the examination of the harmonic components of broadened, specular reflection, and the harmonics of cast shadows.
U.S. patent application Ser. No. 09/967,206 filed Sep. 28, 2001, entitled “Broadened-Specular Reflection and Linear Subspaces for Object Recognition”, the disclosure of which is hereby incorporated by reference, considered harmonic components of broadened, specular reflection in the Phong model.
While the Phong model is mathematically contrived, the Torrance-Sparrow model upon which the present invention is based, is a physically motivated model.
In addition, while the Phong model could be put into a reasonably tractable form by means of standard harmonic analysis, this is an unreasonable goal for the Torrance-Sparrow model. In accordance with the teachings of the present invention, a modified harmonic analysis was used which still enabled an approximation much more readily in the state space of spherical harmonics than in pixel space as graphics people usually do. In the previous invention, the higher harmonics were discarded. In the present invention the higher harmonics are kept in a modified (approximate) form, more approximate the less important the harmonics are.
Most importantly, the present invention makes use of simpler functions where arguments capture more of the angular dependencies than is possible with the usual expansions.
U.S. patent application Ser. No. 09/705,507, filed Nov. 3, 2000, entitled “Lambertian Reflectance and Linear Subspaces”, the disclosure of which is hereby incorporated by reference, considered the relationship between the function that describes the lighting intensity, and the reflectance function that describes how much light an object reflects as a function of its surface normal, under a given lighting condition. Representing these functions as spherical harmonics, it was shown that for Lambertian reflectance the mapping from lighting to reflectance is a convolution with a nearly perfect low-pass filter. High-frequency components of the lighting hardly affect the reflectance function. Therefore, nearly all Lambertian reflectance functions could be modeled as some linear combination of nine spherical harmonic components.
These low-dimensional approximations have two key advantages. First, they are described with few parameters, thus optimization is simpler. Moreover, the images produced under low-frequency lighting are inherently immune to small changes in orientation. A small rotation in the surface normal is equivalent to a small rotation in the light. However, low-frequency light does not appreciably change with a small rotation. The analysis shows this, in that an image produced by an nth order harmonic light component is computed by taking an nth degree polynomial of the components of the surface normal.
While promising, the invention described in U.S. patent application Ser. No. 09/705,507 considered only Lambertian reflectance. Specular reflectance, described in U.S. patent application Ser. No. 09/967,206, can significantly affect how an object appears. For example, a human face may have some oiliness, particularly on the forehead, which reflects light specularly. This is seen as highlights on the object.
While the invention described in U.S. Ser. No. 09/705,507 involved only Lambertian reflection, the Phong model is a simple mathematical rendering of diffuse specular reflectance. The Torrance-Sparrow model underlying the present invention is a fundamental improvement to make the reflectance physically accurate, a necessity for face recognition.
Specular reflectance cannot be approximated well by only considering low-dimensional linear subspaces which capture only low-frequency lighting. Intuitively, the set of possible reflectance functions for a perfect mirror is the same as the set of possible lighting functions. The question remains how to cope with specularity in object recognition using models that will inevitably be imperfect.
Specularities are often thought of as mainly present in spatially localized highlights, which occur when a bright light source is reflected by a shiny object. Such highlights can potentially be identified in the image and processed separately. For example, the algorithms described by Coleman, et al., in “Obtaining 3-Dimensional Shape of Textured and Specular Surfaces Using Four-Source Photometry”, Physics-Based Vision Principles and Practice, Shape Recovery, pp. 180-199 (1992), and by Brelstaff, et al., in “Detecting Specular Reflection Using Lambertian Constraints”, International Conference on Computer Vision, pp. 297-302 (1988), treat specular pixels as statistical outliers that should be discarded. However, specularities due to low-frequency lighting can be spread out throughout the entire image. Since these specularities affect every pixel in the image, their effect must be better understood; they cannot simply be discarded.
Some previous works have considered specular effects also as convolutions. Basri, et al. in NEC Research Institute Technical Report No. 2000-112, entitled “Lambertian Reflectance and Linear Subspaces” (August 2000) as revised in Technical Report No. 2000-172R (December 2000) considered a version of the Phong model in which the specular lobe is a rotationally symmetric shape centered about a peak that is a 180-degree rotation of the direction of the incoming light about the surface normal. In such a model, specular reflection doubles the frequency of the incoming light, and then acts as a convolution. However, this model is heuristic, and not physically based. It does not extend to realistic models.
Ramamoorthi, et al., in “On the Relationship between Radiance and Irradiance: Determining the Illumination From Images of a Convex Lambertian Object”, Journal of the Optical Society of America A, Vol. 18, No. 10, pp. 2448-2459 (2001), considered the effects of arbitrary bi-directional reflectance distribution functions (BRDFs) on the light field. They were able to develop algorithms for recovering the BRDF and/or the lighting given a measured light field and an object with a known 3D structure. Though interesting, these results are not generally applicable to machine vision because, without special sensing devices, the light field is not available. An image of an object is produced by the reflectance function that describes how the surface normals reflect light in one fixed direction. Therefore, an image corresponds to a 2D slice through the light field. For an arbitrary BRDF, the 2D reflectance function for a fixed viewpoint no longer results from a convolution of the lighting.